Multimodal expression for humanoid robots by integration of human speech mimicking and facial color

Multimodal expression is essential for humanoid robots to communicate with people naturally and intelligibly. This paper describes multimodal expression for humanoid robots by mimicking human speech with the ability of expression through “facial colors” . Currently the robot is able to express joy (by turning yellow in the face), anger (red), sadness (blue), and relaxation (green). These colors have been selected according to color psychology. The human speech mimicking is based on prosody extraction of pitch, loudness and temporal information with speech synthesis based on the extracted prosody. The multimodal expression system implemented on Honda ASIMO shows that facial colors improve affective speech recognition by over 15%. In addition, qualitative observations that use speech and facial color with conflicting affective meanings producing complex affection have been reported.